Using Pilot Systems to Execute Many Task Workloads on Supercomputers

Traditionally high-performance computing (HPC) systems have been optimized to support mostly monolithic workloads. The workload of many important scientific applications however, is comprised of spatially and temporally heterogeneous tasks that are often dynamically inter-related. These workloads can benefit from being executed at scale on HPC resources but a tension exists between their resource utilization requirements and the capabilities of HPC system software and HPC usage policies. Pilot systems have successfully been used to address this tension. In this paper we introduce RADICAL-Pilot (RP), a scalable and interoperable pilot system that faithfully implements the Pilot abstraction. We describe its design and characterize the performance of its components, as well as its performance on multiple heterogeneous HPC systems. Specifically, we characterize RP’s task execution component (the RP Agent), which is engineered for optimal resource utilization while maintaining the full generality of the Pilot abstraction.